tpu-inference/k8s: MultiKueue GKE infrastructure for TPU CI - #438

Terraform, Kueue manifest templates and a generator for running TPU CI on
GKE: one CPU-only manager cluster running the Buildkite agent-stack-k8s
controller and Kueue, and one worker cluster per region with TPU node pools
that autoscale from the reservation, joined by MultiKueue over Connect
Gateway.

One input, prod.auto.tfvars: a map of worker clusters, each with its TPU
pools. The pool key is the profile name a pipeline passes and, verbatim, the
node pool, the ClusterQueue and the LocalQueue. terraform apply builds the
clusters, node pools (reservation affinity, image streaming, TPU taint;
a pool with hosts > 1 is a multi-host slice pool with a placement policy
carrying the topology, one node per host, scaled atomically), the regional
cache buckets with their PersistentVolumes and claims, IAM (dedicated node
service accounts, Workload Identity for the workload service account and the
Kueue controllers, cross-project Artifact Registry read), External Secrets,
the JobSet operator and Kueue itself. scripts/generate_manifests.py renders
everything Kubernetes-side from the same file - cohorts and ResourceFlavors
per accelerator, ClusterQueues and LocalQueues per pool (a worker's quota is
its own capacity, the manager's is the sum over the workers that have the
profile), MultiKueue fleet and AdmissionChecks, per-worker RBAC, the
workload ServiceAccount and the cache volumes - into generated/, which is
committed so a diff after an unrelated change means drift.

Only google.com/tpu is under Kueue quota; cpu and memory are ignored by
admission (quotaCheckStrategy: IgnoreUndeclared) and left to the scheduler.
The v6e worker in southamerica-west1-a carries two profiles on one 18-chip
reservation: v6e-1-1x1 (ten chips, two kept warm) and v6e-8-2x4 (one node).

Validated end to end from tpu-inference: five full per-push suites (kube-dev
builds 255-260), and a four-host JobSet slice on this infrastructure (build
272) - the multi-host pool support is kept; the pool itself is not declared.

scripts/deploy_manifests.sh applies that committed output rather than
regenerating over it, so a shared cluster only ever sees manifests that went
through review; it renders into a scratch directory first purely to prove the
committed ones are current, and refuses to apply if they are not. Each cluster
is addressed by an explicit --context captured from its own get-credentials,
in a kubeconfig of the script's own, and every apply is preceded by a kubectl
diff and a confirmation. It still only creates and updates - a queue dropped
from the tfvars stays in the cluster until someone removes it.

The Kueue controller Deployment is patched from kueue.tf rather than from that
script. The Deployment is the chart's, so any release that re-renders it - a
kueue_version bump, an edit to manager-config.yaml - drops the gcp-auth-plugin
initContainer and the manager silently loses the credentials it dispatches to
workers with. Keyed on the release revision, the patch goes back on whenever
that happens; chart 0.19.0 exposes no values that would carry it.

initial_node_count is ignored on the worker TPU pools, as it already was on
the manager's system pool. It is a create-time field GKE reports as whatever
the pool has scaled to since, so left tracked it drifts on its own and any
edit to min_nodes reads as a change to it - forcing the pool to be destroyed
and rebuilt when all that was wanted is a new autoscaling floor.